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Mail Stop Appeal Brief - Patents 
Commissioner for Patents 
P.O. Box 1450 
Alexandria, VA 2231 3-1450 

Dear Sir: 

This is an appeal from the Final Action, dated September 7, 2007, for the 
above identified patent application. Appellants submit that this Appeal Brief is 
being timely filed before the final deadline of February 7, 2008. Please charge 
the Appeal Brief fee of $510 to deposit account no. 08-2025. 

REAL PARTY IN INTEREST 

The real party in interest to the present application is Hewlett-Packard 
Development Company, LP, a limited partnership established under the laws of 
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the State of Texas and having a principal place of business at 20555 S.H. 249 
Houston, TX 77070, U.S.A. (hereinafter "HPDC"). HPDC is a Texas limited 
partnership and is a wholly-owned affiliate of Hewlett-Packard Company, a 
Delaware Corporation, headquartered in Palo Alto, CA. The general or 
managing partner of HPDC is HPQ Holdings, LLC. 

RELATED APPEALS AND INTERFERENCES 

Appellants submit that there are no other prior and pending appeals, 
interferences or judicial proceedings which may be related to, directly affect or be 
directly affected by or have a bearing on the Board's decision in the pending 
appeal. 

STATUS OF CLAIMS 

Claims 1-20 are currently pending. Claims 1-20 stand rejected and are the 
subject of this Appeal and are reproduced in the accompanying Claims Appendix. 

STATUS OF AMENDMENTS 

No Amendment After Final Rejection has been entered. 

SUMMARY OF CLAIMED SUBJECT MATTER 

The invention described and claimed in the present application relates to 
dynamic control of resource usage in a multimodal system. 

The present invention teaches a method of dynamically controlling usage of 
a resource by task entities respectively involved in processing different input 
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modalities, wherein the relative average actual or allocated usage of the resource 
by the task entities is dynamically adjusted according to one or more of the 
following: actual usage of the different modalities by a user; confidence in the 
results of processing of each of the modalities; pragmatic information on mode 
usage. Pragmatic information on mode usage provides a measure of how the 
target application is set up to use input from different modes-in other words, 
whether input from one modality is more important or useful than that from 
another modality, at least in the current application context (Page 2, line 30 - 
Page 3, line 9). 

Claim 1 of the present invention is directed to a method of dynamically 
controlling usage of a resource by data-processing entities [65, 66, 67, 68 of FIG. 
2] of a multimodal system that is arranged to receive user input in multiple input 
modalities for use in combination by an application being run by the multimodal 
system, the data-processing entities being involved in processing different input 
modalities in a data-processing device, the method comprising: receiving inputs 
regarding: input mode usage [71] by a user of the data processing device, modal 
requirements [71] of a dialogue manager and an application or service, and/or 
confidence in a recognition process [71] for each modality [input to 65, input to 67 
of FIG. 2] at a bandwidth moderator [70]; determining a target relative usage of a 
data-processing resource; wherein a relative average actual or allocated usage 
of the resource by the data-processing entities is dynamically allocated by said 
bandwidth moderator [70] according to one or more of the following: 

actual usage of the different input modalities [First Modality, Second Modality, 
FIG. 2] by the user of the device; 
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confidence in the results of processing carried out in respect of each of the 
input modalities; 

pragmatic [62 of FIG. 2, 30 of FIG.3] information on input modality usage; 

and 

processing at least one of the input modalities using the resource as 
dynamically allocated by said bandwidth moderator [70]. 

Claim 10 of the present invention is directed to a multimodal system 
comprising: one or more processors for running application software; input 
apparatus for receiving user inputs in multiple different input modalities [Speech, 
Gestures] for use by said application software; data-processing entities [65, 66, 
67, 68 of FIG. 2] respectively involved in processing data in respect of different 
input modalities; a limited resource arranged to be used by the data-processing 
entities; and a moderator [70] for dynamically adjusting a relative average actual 
or allocated usage of the resource by the data-processing entities in dependence 
on one or more of the following: 

actual usage of the different input modalities by a user; 

confidence in the results of processing carried out in respect of each of the 
input modalities; 

pragmatic [62 of FIG. 2, 30 of FIG.3] information on input modality usage. 
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GROUNDS OF REJECTION TO BE REVIEWED ON APPEAL 

Issue 1: Whether claims 1, 3, 9-10, 12, and 18-20 are patentable under 35 U.S.C. 
§ 103(a) over Maes et al., U.S. Patent No. 6,964,023, (hereinafter, Maes) in view 
of Suhm et a/., "Multimodal Error Correction for Speech User Interfaces, 2001," 
(hereinafter, Suhm)? 

Issue 2: Whether claims 2, 4-8, 11, and 13-17 are patentable under 35 U.S.C. § 
103(a) over Maes in view of Suhm and further in view of Bridger et al. (WO 
01/35575 A2) (hereinafter, Bridger)? 

ARGUMENT 

Issue 1: Whether claims 1, 3, 9-10, 12, and 18-20 are patentable under 35 U.S.C. 
§ 103(a) over Maes et al., U.S. Patent No. 6,964,023, (hereinafter, Maes) in view 
of Suhm et al., "Multimodal Error Correction for Speech User Interfaces, 2001," 
(hereinafter, Suhm)? 

In the final Office Action of September 7, 2007, the Examiner rejects 
Claims 1, 3, 9-10, 12, and 18-20 under 35 U.S.C. 102(e) as being unpatentable 
over Maes in view of Suhm. Appellants respectfully disagree with the Examiner's 
rejection and request that the rejection be overturned on appeal. 

Maes teaches systems and methods provided for performing focus 
detection, referential ambiguity resolution and mood classification in accordance 
with multi-modal input data, in varying operating conditions, in order to provide an 
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effective conversational computing environment for one or more users. (Maes 
Abstract). 

The Suhm article teaches two approaches to make correction by repetition 
effective: switching modality for repetitions and correlating the correction input 
with repair context . In cross-modal repair, the user corrects with a different 
modality than used for the primary input. (Please see page 68, first full 
paragraph). 

On page 5 of the Action, the Examiner states, "receiving inputs regarding: 
input mode usage by a user of the data processing device, modal requirements 
of a dialogue manager and an application or service, and/or confidence in a 
recognition process for each modality at a bandwidth moderator" of Appellants' 
claim 1 is disclosed by Maes in that the conversational resource manager 
receives inputs regarding engaged input modalities and associated dialog 
applications, modality capabilities/states, and network path delay. 

However, Maes does not disclose a bandwidth moderator. Maes 
discloses "bandwidth" merely in the context of describing an acoustic front-end 
as in "Frames are advanced every 10 msec to obtain succeeding acoustic 
vectors. Note that other acoustic front-ends with other frame sizes and sampling 
rates/signal bandwidths can also be employed" (Column 12, lines 9-11, of Maes). 
There is no teaching in Maes of "receiving inputs regarding: input mode usage by 
a user of the data processing device, modal requirements of a dialogue manager 
and an application or service, and/or confidence in a recognition process for each 
modality at a bandwidth moderator" as recited in claim 1 and "a moderator for 
dynamically adjusting a relative average actual or allocated usage of the 
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resource by the data-processing entities..." as recited in claim 10. 

It is noteworthy that in rejecting certain dependent claims, on page 7 of the 
Action, lines 6-7, the Examiner states, "Maes in view of Suhm does not explicitly 
suggest resources comprising communication bandwidth and memory." This 
further reinforces Appellants' position that Maes does not teach "receiving inputs 
regarding: input mode usage by a user of the data processing device, modal 
requirements of a dialogue manager and an application or service, and/or 
confidence in a recognition process for each modality at a bandwidth moderator" 
as recited in claim 1 and "a moderator for dynamically adjusting a relative 
average actual or allocated usage of the resource by the data-processing entities 
in dependence on one or more of the following..." as recited in claim 10. 

Further, the examiner concedes that Maes does not specifically suggest 
receiving a confidence score in a recognition process for resource allocation (last 
paragraph, page 5 of the Action), and cites Suhm as reciting using confidence 
scores to identify recognition errors. However, in addition to Suhm's teaching 
neither "receiving inputs regarding: input mode usage by a user of the data 
processing device, modal requirements of a dialogue manager and an 
application or service, and/or confidence in a recognition process for each 
modality at a bandwidth moderator," as recited in claim 1 , nor "a moderator for 
dynamically adjusting a relative average actual or allocated usage of the 
resource by the data-processing entities in dependence on one or more of the 
following..." as recited in claim 10, Suhm teaches away from the invention 
because Suhm questions whether confidence scores can facilitate error 
correction: 
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Words with low confidence scores are tagged as possible 
recognition errors. Since confidence scores themselves are 
not reliable, these tags may be incorrect . (Emphasis added ), 
(Last paragraph beginning on page 74, of Suhm). 

In addition, this study investigated the important question of 
whether system-initiated error detection can improve dictation 
performance. Our results question the common belief (among 
many researchers in the speech recognition community) that 
confidence scores can facilitate error detection . While this 
result is limited to the present implementation of confidence 
scores, anecdotal comments suggest that other 
implementations by developers of commercial dictation systems 
have failed to realize a gain as well. (Emphasis added ), (Last 
paragraph beginning on page 94, of Suhm). 



Accordingly, the significant gap left by Maes is not bridged by Suhm. 
Therefore, Appellants respectfully submit that independent claims 1 and 10 are 
patentably unobvious over Maes in view of Suhm. Appellants request that the 
rejection be reversed on appeal. 

Claims 3, 9, and 19 depend from claim 1 and, therefore, are patentably 
unobvious at least for the reasons stated above. Claims 12, 18, and 20 depend 
from claim 1 and, therefore, are patentably unobvious at least for the reasons 
stated above. Appellants request that their rejection be reversed on appeal. 



Issue 2: Whether claims 2, 4-8, 11, and 13-17 are patentable under 35 U.S.C. § 
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103(a) over Maes in view of Suhm and further in view of Bridger et al. (WO 
01/35575 A2) (hereinafter, Bridger)? 

In the final Office Action of September 7, 2007, the Examiner rejects 
claims 2, 4-8, 11, and 13-17, as being unpatentable over Maes in view of Suhm 
and further in view of Bridger. Appellants respectfully disagree with the 
Examiner's rejection and request that the rejection be overturned on appeal. 

Bridger is nothing more than a channel router. It is not concerned with "a 
multimodal system that is arranged to receive user input in multiple input 
modalities for use in combination by an application being run by the multimodal 
system" as set forth in claim 1 , for example. 

Anyway, even if it were obvious to stuff Bridger's channel router into Maes' 
hardware, the limitation underlined in the preceding section is not met by the 
cited art. 

Claims 2, and 4-8 depend from claim 1, and claims 11, and 13-17, depend 
from claim 10. Therefore, these claims are patentably unobvious at least for the 
reasons stated above. Appellants request that their rejection be reversed on 
appeal. 
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CONCLUSION 

For the reasons advanced above, Appellants respectfully contend that each 
claim is patentable. Therefore, reversal of all rejections is courteously solicited. 

The Commissioner is authorized to charge any additional fees, which may 
be required or credit overpayment to deposit account no. 12-0415. In particular, 
if this response is not timely filed, then the Commissioner is authorized to treat 
this response as including a petition to extend the time period pursuant to 37 
CFR § 1.136 (a) requesting an extension of time of the number of months 
necessary to make this response timely filed and the petition fee due in 
connection therewith may be charged to deposit account no. 12-0415. 

I hereby certify that this Respectfully submitted, 
correspondence is being transmitted 
to the United States Patent and 

Trademark Office via electronic filing /Amit Singh/ 

on Amit Singh 



Attorney for Appellants 



(Date of Transmittance) 



February 4. 2008 



Reg. No. 54451 
LADAS & PARRY 



Krista Celentano 



5670 Wilshire Blvd., Suite 2100 
Los Angeles, CA 90036 



(Name of Person Transmitting) 



(323)934-2300 



/Krista Celentano/ 



Signature 
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1 . A method of dynamically controlling usage of a resource by data-processing 
entities of a multimodal system that is arranged to receive user input in multiple 
input modalities for use in combination by an application being run by the 
multimodal system, the data-processing entities being involved in processing 
different input modalities in a data-processing device, the method comprising: 

receiving inputs regarding: input mode usage by a user of the data 
processing device, modal requirements of a dialogue manager and an application 
or service, and/or confidence in a recognition process for each modality at a 
bandwidth moderator; 

determining a target relative usage of a data-processing resource; 

wherein a relative average actual or allocated usage of the resource by the 
data-processing entities is dynamically allocated by said bandwidth moderator 
according to one or more of the following: 

actual usage of the different input modalities by a the user of the 

device; 

confidence in the results of processing carried out in respect of each of 
the input modalities; 

pragmatic information on input modality usage; and 
processing at least one of the input modalities using the resource as 
dynamically allocated by said bandwidth moderator. 

2. A method according to claim 1, wherein the resource is communication 
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bandwidth. 

3. A method according to claim 1 , wherein the resource is processing power. 

4. A method according to claim 1 , wherein the resource is memory. 

5. A method according to claim 1 applied to each of two separate resources 
each used by different respective entities of said different input modalities, the 
adjustment of the relative usage by the different modalities of the two resources 
being independent of each other. 

6. A method according to claim 1 applied to each of two separate resources 
each used by different respective entities of said different input modalities, the 
adjustment of the relative usage by the different modalities of the two resources 
being jointly controlled. 

7. A method according to claim 1, wherein said resource is used by multiple 
data-processing entities for each modality, the relative usage of the resource 
being first adjusted between modalities and then between data-processing 
entities in the same modality. 

8. A method according to claim 1 , wherein said resource is used by multiple 
data-processing entities for each modality, the relative usage of the resource 
being first adjusted between different groups of equivalent data-processing 
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entities of different modalities and then between data-processing entities of the 
same group. 

9. A method according to claim 1, wherein adjustment of the relative usage of 
the resource allocation is effected by one of: controlling operation of the 
data-processing entities to adjust their output to the resource; controlling the flow 
of output from the data-processing entities to the resource; controlling the 
allocation of the resource between the data-processing entities. 

1 0. A multimodal system comprising; 

one or more processors for running application software; 
input apparatus for receiving user inputs in multiple different input modalities 
for use by said application software; 

data-processing entities respectively involved in processing data in respect 
of different input modalities; 

a limited resource arranged to be used by the data-processing entities; and 
a moderator for dynamically adjusting a relative average actual or allocated 
usage of the resource by the data-processing entities in dependence on one or 
more of the following: 

actual usage of the different input modalities by a user; 
confidence in the results of processing carried out in respect of each of 
the input modalities; 

pragmatic information on input modality usage. 
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11. A system according to claim 10, further comprising a respective additional 
data-processing entity associated with each said input modality, and a 
communications system arranged to intercommunicate the data-processing 
entities associated with the same input modality; said limited resource being 
communication bandwidth provided by said communications system. 

12. A system according to claim 10, wherein the data-processing entities 
comprise a shared processing system and said limited resource is the processing 
power provided by this processing system. 

13. A system according to claim 10, wherein the data-processing entities 
comprise a shared memory unit and said limited resource is the memory 
provided by the memory unit. 

14. A system according to claim 10, further comprising further data- 
processing entities involved in processing respective ones of said input 
modalities, a further limited resource arranged to be used by said further 
data-processing entities, and a further moderator for dynamically adjusting the 
relative average actual or allocated usage of the resource by the further 
data-processing entities; the operation of the two moderators being independent 
of each other. 

15. A system according to claim 10, further comprising further data-processing 
entities involved in processing respective ones of said input modalities, a further 
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limited resource arranged to be used by said further data-processing entities, and 
a further moderator for dynamically adjusting the relative average actual or 
allocated usage of the resource by the further data-processing entities; the 
moderators being arranged to operate in a coordinated manner. 

16. A system according to claim 10, further comprising further data-processing 
entities involved in processing respective ones of said input modalities, the 
further data-processing entities also being arranged to use said resource and the 
moderator being arranged first to adjust relative usage of said resource between 
modalities and then between data-processing entities in the same modality. 

1 7. A system according to claim 10, further comprising further data- 
processing entities involved in processing respective ones of said input 
modalities, the further data-processing entities also being arranged to use said 
resource and the moderator being arranged first to adjust relative usage of said 
resource between different groups of equivalent data-processing entities of 
different modalities and then between data-processing entities of the same 
group. 

18. A system according to claim 10, wherein the moderator is arranged to effect 
adjustment of the relative usage of the resource by one of: controlling operation 
of the data-processing entities to adjust their output to the resource; controlling 
the flow of output from the data-processing entities to the resource; controlling 
the allocation of the resource between the data-processing entities. 
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19. A method according to claim 1 wherein the relative average actual or 
allocated usage of the resource by the data-processing entities is dynamically 
allocated by said bandwidth moderator according all of the following: 

(i) actual usage of the different input modalities by the user of the 

device; 

(ii) confidence in the results of processing carried out in respect of 
each of the input modalities; 

(iii) pragmatic information on input modality usage, 

but wherein only one of the three items (i) to (iii) need exist for dynamic 
allocation of the resource to occur. 

20. A system according to claim 10 wherein the relative average actual or 
allocated usage of the resource by the data-processing entities is dynamically 
allocated by said moderator according all of the following: 

(i) actual usage of the different input modalities by the user of the 

device; 

(ii) confidence in the results of processing carried out in respect of 
each of the input modalities; 

(iii) pragmatic information on input modality usage, 

but wherein only one of the three items (i) to (iii) need exist for dynamic 
allocation of the resource to occur. 
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Evidence Appendix 



No evidence is being submitted. 
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Related Proceedings Appendix 



No copies of decisions rendered in related proceedings are being submitted. 



